{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "wzY3w4DFMwJx"
      },
      "source": [
        "##### Copyright 2018 The TensorFlow Authors.\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "rAl-m3CJM9f2"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\"); { display-mode: \"form\" }\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "dm-eURywwxqM"
      },
      "source": [
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://www.tensorflow.org/tutorials/tensorflow_text/intro\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003eView on TensorFlow.org\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/text/blob/master/examples/intro.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/text/blob/master/examples/intro.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView source on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca href=\"https://storage.googleapis.com/tensorflow_docs/text/examples/intro.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /\u003eDownload notebook\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "bkDse1F8QfLC"
      },
      "source": [
        "# TF.Text\n",
        "\n",
        "## Introduction\n",
        "\n",
        "TensorFlow Text provides a collection of text related classes and ops ready to use with TensorFlow 2.0. The library can perform the preprocessing regularly required by text-based models, and includes other features useful for sequence modeling not provided by core TensorFlow.\n",
        "\n",
        "The benefit of using these ops in your text preprocessing is that they are done in the TensorFlow graph. You do not need to worry about tokenization in training being different than the tokenization at inference, or managing preprocessing scripts."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "5vzUaGC_QimJ"
      },
      "source": [
        "## Eager Execution\n",
        "\n",
        "TensorFlow Text requires TensorFlow 2.0, and is fully compatible with eager mode and graph mode.\n",
        "\n",
        "---\n",
        "\n",
        "Note: On rare occassions, this import may fail looking for the TF library. Please reset the runtime and rerun the pip install below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "o3iZ_QvCzuLD"
      },
      "outputs": [],
      "source": [
        "!pip install tensorflow-text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "MjeiCYH2HgLG"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "import tensorflow_text as text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "zBz-AF1lQwO0"
      },
      "source": [
        "## Unicode\n",
        "\n",
        "Most ops expect that the strings are in UTF-8. If you're using a different encoding, you can use the core tensorflow transcode op to transcode into UTF-8. You can also use the same op to coerce your string to structurally valid UTF-8 if your input could be invalid.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "mup7OvnUQyrk"
      },
      "outputs": [],
      "source": [
        "docs = tf.constant([u'Everything not saved will be lost.'.encode('UTF-16-BE'), u'Sad☹'.encode('UTF-16-BE')])\n",
        "utf8_docs = tf.strings.unicode_transcode(docs, input_encoding='UTF-16-BE', output_encoding='UTF-8')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "JhF9xiL4RdGZ"
      },
      "source": [
        "## Tokenization\n",
        "\n",
        "Tokenization is the process of breaking up a string into tokens. Commonly, these tokens are words, numbers, and/or punctuation.\n",
        "\n",
        "The main interfaces are `Tokenizer` and `TokenizerWithOffsets` which each have a single method `tokenize` and `tokenize_with_offsets` respectively. There are multiple tokenizers available now. Each of these implement `TokenizerWithOffsets` (which extends `Tokenizer`) which includes an option for getting byte offsets into the original string. This allows the caller to know the bytes in the original string the token was created from.\n",
        "\n",
        "All of the tokenizers return RaggedTensors with the inner-most dimension of tokens mapping to the original individual strings. As a result, the resulting shape's rank is increased by one. Please review the ragged tensor guide if you are unfamiliar with them. [ragged_tensors](https://www.tensorflow.org/guide/ragged_tensors)\n",
        "\n",
        "### WhitespaceTokenizer\n",
        "\n",
        "This is a basic tokenizer that splits UTF-8 strings on ICU defined whitespace characters (eg. space, tab, new line).\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "pj6QRMJ_RbTf"
      },
      "outputs": [],
      "source": [
        "tokenizer = text.WhitespaceTokenizer()\n",
        "tokens = tokenizer.tokenize(['everything not saved will be lost.', u'Sad☹'.encode('UTF-8')])\n",
        "print(tokens.to_list())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "bTuPjpyrGwKK"
      },
      "source": [
        "### UnicodeScriptTokenizer\n",
        "\n",
        "This tokenizer splits UTF-8 strings based on Unicode script boundaries. The script codes used correspond to International Components for Unicode (ICU) UScriptCode values. See: http://icu-project.org/apiref/icu4c/uscript_8h.html\n",
        "\n",
        "In practice, this is similar to the `WhitespaceTokenizer` with the most apparent difference being that it will split punctuation (USCRIPT_COMMON) from language texts (eg. USCRIPT_LATIN, USCRIPT_CYRILLIC, etc) while also separating language texts from each other."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "QIAu7D7JG6xC"
      },
      "outputs": [],
      "source": [
        "tokenizer = text.UnicodeScriptTokenizer()\n",
        "tokens = tokenizer.tokenize(['everything not saved will be lost.', u'Sad☹'.encode('UTF-8')])\n",
        "print(tokens.to_list())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Lzuqb8xkG7Cp"
      },
      "source": [
        "### Unicode split\n",
        "\n",
        "When tokenizing languages without whitespace to segment words, it is common to just split by character, which can be accomplished using the [unicode_split](https://www.tensorflow.org/api_docs/python/tf/strings/unicode_split) op found in core."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "GCKTl2mzdZIo"
      },
      "outputs": [],
      "source": [
        "tokens = tf.strings.unicode_split([u\"仅今年前\".encode('UTF-8')], 'UTF-8')\n",
        "print(tokens.to_list())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "0yGs1mWkeOD8"
      },
      "source": [
        "### Offsets\n",
        "\n",
        "When tokenizing strings, it is often desired to know where in the original string the token originated from. For this reason, each tokenizer which implements `TokenizerWithOffsets` has a *tokenize_with_offsets* method that will return the byte offsets along with the tokens. The start_offsets lists the bytes in the original string each token starts at, and the end_offsets lists the bytes immediately after the point where each token ends.  Note: the start offsets are inclusive and the end offsets are exclusive."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "8yzUormbiQwz"
      },
      "outputs": [],
      "source": [
        "tokenizer = text.UnicodeScriptTokenizer()\n",
        "(tokens, start_offsets, end_offsets) = tokenizer.tokenize_with_offsets(['everything not saved will be lost.', u'Sad☹'.encode('UTF-8')])\n",
        "print(tokens.to_list())\n",
        "print(start_offsets.to_list())\n",
        "print(end_offsets.to_list())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "iO-tRdH72Vl9"
      },
      "source": [
        "### TF.Data Example\n",
        "\n",
        "Tokenizers work as expected with the tf.data API. A simple example is provided below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "0MCqcHXg3gtp"
      },
      "outputs": [],
      "source": [
        "docs = tf.data.Dataset.from_tensor_slices([['Never tell me the odds.'], [\"It's a trap!\"]])\n",
        "tokenizer = text.WhitespaceTokenizer()\n",
        "tokenized_docs = docs.map(lambda x: tokenizer.tokenize(x))\n",
        "iterator = iter(tokenized_docs)\n",
        "print(next(iterator).to_list())\n",
        "print(next(iterator).to_list())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "cAFvE0bDi5mq"
      },
      "source": [
        "## Other Text Ops\n",
        "\n",
        "TF.Text packages other useful preprocessing ops. We will review a couple below.\n",
        "\n",
        "### Wordshape\n",
        "\n",
        "A common feature used in some natural language understanding models is to see if the text string has a certain property. For example, a sentence breaking model might contain features which check for word capitalization or if a punctuation character is at the end of a string.\n",
        "\n",
        "Wordshape defines a variety of useful regular expression based helper functions for matching various relevant patterns in your input text. Here are a few examples."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "1HK5KOj5IJ5m"
      },
      "outputs": [],
      "source": [
        "tokenizer = text.WhitespaceTokenizer()\n",
        "tokens = tokenizer.tokenize(['Everything not saved will be lost.', u'Sad☹'.encode('UTF-8')])\n",
        "\n",
        "# Is capitalized?\n",
        "f1 = text.wordshape(tokens, text.WordShape.HAS_TITLE_CASE)\n",
        "# Are all letters uppercased?\n",
        "f2 = text.wordshape(tokens, text.WordShape.IS_UPPERCASE)\n",
        "# Does the token contain punctuation?\n",
        "f3 = text.wordshape(tokens, text.WordShape.HAS_SOME_PUNCT_OR_SYMBOL)\n",
        "# Is the token a number?\n",
        "f4 = text.wordshape(tokens, text.WordShape.IS_NUMERIC_VALUE)\n",
        "\n",
        "print(f1.to_list())\n",
        "print(f2.to_list())\n",
        "print(f3.to_list())\n",
        "print(f4.to_list())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "8zrnuXTc3ziR"
      },
      "source": [
        "### N-grams \u0026 Sliding Window\n",
        "\n",
        "N-grams are sequential words given a sliding window size of *n*. When combining the tokens, there are three reduction mechanisms supported. For text, you would want to use `Reduction.STRING_JOIN` which appends the strings to each other. The default separator character is a space, but this can be changed with the string_separater argument.\n",
        "\n",
        "The other two reduction methods are most often used with numerical values, and these are `Reduction.SUM` and `Reduction.MEAN`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "czIf9HcoIquB"
      },
      "outputs": [],
      "source": [
        "tokenizer = text.WhitespaceTokenizer()\n",
        "tokens = tokenizer.tokenize(['Everything not saved will be lost.', u'Sad☹'.encode('UTF-8')])\n",
        "\n",
        "# Ngrams, in this case bi-gram (n = 2)\n",
        "bigrams = text.ngrams(tokens, 2, reduction_type=text.Reduction.STRING_JOIN)\n",
        "\n",
        "print(bigrams.to_list())"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "last_runtime": {
        "build_target": "//experimental/socrates:notebook",
        "kind": "private"
      },
      "name": "intro.ipynb",
      "private_outputs": true,
      "provenance": [],
      "toc_visible": true,
      "version": "0.3.2"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
